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ABSTRACT 

The evaluation of this pilot of an epidemiology course conducted online and delivered 
across Canada was based on four main criteria: design, content, process and outcomes. 
Data was collected through seven sources: participant online survey results, post-pilot 
workshop feedback, four focus groups, telephone interviews with participants, interviews 
with course developers, examination of online materials, and analysis of log files 
generated by the web server. The pilot course had the following outcomes: 

Findings on the delivery showed that the course took much more time than estimated; 
the online environment was challenging/frustrating for some; there were technical 
glitches; discussion boards were not regularly used; interaction with instructors was 
minimal; feedback from instructors was slow in coming; the short development time led 
to errors/ mismatches between content and assessment; and the high student/teacher 
ratio of 1:48 made it difficult to provide timely feedback. 

Keywords: Distance learning, case study, health education, interaction, 

Web-based learning 

INTRODUCTION 

Health Canada's Centre for Surveillance Coordination used Internet-based distance 
learning to deliver a course in epidemiology. This pilot project was an initiative to 
increase the skills of nursing and other staff in local public health departments and 
regional health authorities across Canada. Building an online course allowed the 
Surveillance Centre to reach geographically dispersed public health workers, and to 
further employees' access to continuing professional development. 

The pilot course in epidemiology was developed by ZeddComm Inc. and Memorial 
University of Newfoundland (MUN). The course goals included the following: 

> create a prototype course promoting active learning; 

> use discussion as part of the learning process; 
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> make the course accessible at the workplace; 

> use self-evaluation as a type of assessment; and 

> give students control over pace and time spent on the course. 

MUN faculty used the WebCT learning management system to teach 48 participants, 43 of 
whom were active throughout the pilot. 

The decision was made to use the Internet as the mode of delivery because the potential 
students for this project were spread across Canada. The New Brunswick Distance 
Education Network and the Consortium for Information Technology (CITE) provided an 
evaluation of instructional approaches for delivery. Another useful document in making 
the decision to go online was a University of Ottawa paper, entitled Internet-based 
distance learning in epidemiology and public health for health surveillance, which 
concluded that this initiative would not duplicate existing courses and they also 
recommended that Canadian content be included in any new offerings of courses. 

The following decisions were made as a result of the papers that were produced and the 
consultations that were held: 

> Course would be web-based, interactive, self-paced, and take 
approximately 20 hours to complete. 

> The first module developed would be an Introduction to Epidemiology 
course. 

> The target audience for the pilot would be public health professionals in a 
variety of front line and management positions. 

> The prototype developed for this pilot would serve as a model for future 
courses. 

Three goals were set for the Skills Enhancement for Health Surveillance initiative: 

> Training should increase the awareness of basic principles of epidemiology, 
as well as 

> An appreciation for the value of those principles. 

> New knowledge and skills acquired will be transferred to the workplace. 
DESCRIPTION OF THE INTRODUCTION TO EPIDEMIOLOGY COURSE 

After a formal evaluation, WebCT was the delivery platform chosen for this pilot. The 
course content was divided into three units: 

Unit 1: covered basic epidemiological concepts - five lessons 
Unit 2: dealt with health status assessment plus form and presentation of 
epidemiological data - three lessons 

Unit 3: looked at the types of epidemiological studies plus methods used to 

investigate disease - three lessons 

Each unit had several features: 

> Pre-test and Post-test (multiple choice or matching questions scored by 
WebCT); 

> Content section; 

> Learning objectives; and 

> Self-assessment exercise. 

The left menu bar provided shortcuts to Course Information, Discussions, and Learning 
Exercises. The top menu bar provided links to My WebCT, Resume Course, Course Map, 
and WebCT Help Files. These bars were visible and accessible at all times to the learner. 
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The content areas provided text, illustrations, graphs, and pictures as well as web links on 
the topic being covered. Learning objectives were always second on the list after the 
introduction to the content. Five lessons had interview audio files available in either 
RealAudio or Win Media. 

Bulletin Board Discussion Activities provided concise topic-related information. Students 
were to post answers to questions for all to see. Participants could link to the Bulletin 
Board Discussion within the Bulletin Board Discussion question or through a link on the 
Homepage. General discussion areas were provided through Course Feedback and Main 
links. Nine of the eleven lessons contained learning exercises that required the student to 
respond to an essay type question. These responses were graded manually by the 
instructor, who also provided feedback. Participants received up to ten points for 
submissions. 

Multiple choice quizzes were the medium chosen for self-assessment activities. This 
provided instant feedback to the learner. Scores were not tallied and explanations for 
correct answers were pre-configured and immediately available. Students could do self- 
assessments as often as they pleased and results were not saved. Technical help was 
available through a toll-free number. Help using WebCT itself was available online within 
the course. No training was provided prior to beginning the course. 

METHODOLOGY 

A variety of methods for evaluation was employed in order to provide for triangulation. 
The process was valued equally with the product. Ongoing feedback throughout the pilot 
provided opportunity for modifications to be made to improve the experience for the 
students. Four criteria were set for evaluation: 

> 

> Design: suitability of online learning environment, participant expectations, 
module design, and assessments. 

> Content: appropriateness, interest/ motivation to learn, difficulty, amount of 
material, value of resources/links, orientation to online environment. 

> Process: pacing, support, feedback, technical skill deficiencies, technical 
glitches, effective use of time, access to technology, interaction. 

> Outcome: goals met, real world application, understanding the real world, skill 
acquisition. 

Flashlight Current Student Inventory (CSI) was used to develop survey instruments. The 
online participant survey consisted of 37 Likert-type questions, 2 multiple choice and 3 
open-ended questions. The focus group protocol and the telephone interview guide were 
also prepared using CSI. 

Participant Information 

Developers expected to have 20 to 25 participants. However, 48 learners enrolled and 43 
of them remained active for the duration. Not all participants completed the course or 
engaged fully in the content. All participants were public health professionals. All the 
Canadian provinces and territories were represented. 

Data Collection Procedures 

> Online survey; 

> Four focus groups prior to the Health Canada two-day workshop; 

> Two-day workshop; 

> Course developer interviews; and 

> Telephone interviews with those who could not attend focus groups. 
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DISCUSSION AND RECOMMENDATIONS 



Design 

The course home page needed a welcome screen, and art work could have been more 
professional rather than the clip art that was used. There should be clear, concise 
directions on how to navigate the course. Color coding could be more effectively used and 
Units should be appropriately titled rather than simply numbered. The screen appeared 
cluttered and should be reorganized and simplified. The introduction to each unit should 
state the amount of time expected to complete as well as directions to the assessment 
activities. Use blue for hypertext links. 

The concepts were successfully integrated with media in the course design. Information 
cards were valued and could be part of the course materials package. Audio files were not 
successfully utilized by students, but are a good strategy. Therefore they should be 
revised. 

The amount of scrolling required was an issue with learners. Hypertext anchors can be 
used more effectively to ease navigation. Use of the back button was not fully explained. 
Screen shots of external links could reduce frustration. 

There should be a list of learning outcomes for the course as a whole. A course syllabus 
should be added so students can orient themselves more fully. Learning outcomes should 
require more cognitive involvement from the participants. Assessments and learning 
activities should more closely relate to learning objectives. Multiple choice course 
assessments do not require higher level understanding of course material. 

Prior to starting the course, students should have time for training on how to make best 
use of the online learning environment. Students did not make good use of online help 
available in WebCT, so other means of providing support should be investigated. 

System configuration information should be more easily discovered. Use of an online 
system check would have been appreciated by learners so that they did not have to figure 
out if they had the right resources required. 

WebCT is a good platform for the course. However, developers could look at additional 
options for delivery, such as CD-ROMs. PDF files could be used within WebCT to allow 
students to download, save, and print offline. 

Improvements suggested for the course development cycle: 

> Standardized format for all future modules; 

> Six-month time frame to produce learning activities; and 

> Pilot test modules with small group of learners, with half of the group 
being new to online learning. 



Content 

It is recommended that 25 hours be the time allotted for participants to spend on the 
course. Participants took up to 40 hours to complete the pilot. Therefore, content should 
be reduced. The learning objectives were adequately supported by the content and were 
sequenced effectively. 

Historical content should remain, but not be tested. Pre- and post-tests caused anxiety 
for some participants. However, it was recommended that they be continued, although 
steps should be taken to reduce stress for the learner. 
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Process 

Learning exercises were seen as relevant to the participants but should be tailored to a 
variety of positions held in public health. Although discussion board activities did not 
activate as much student-to-student interaction as was hoped, they should be included in 
future modules to encourage growth of an online community. Both synchronous and 
asynchronous discussions could be explored. Face-to-face seminars at regional centres 
could also act as catalysts for student interaction. 

Getting prompt feedback from instructors was an issue. This could be aided by limiting 
enrollment to a 1:20 student/teacher ratio as the enrollment in the pilot was 1:43. 
During the pilot, instructors were still working on course content. This was addressed 
earlier when the recommendation was made that developers be given six months to 
prepare online modules. 

Access to technology caused problems. Informing students of technical requirements up- 
front would help. There also needs to be an investment in technology made by employers 
and government agencies. 

Technical support is critical, especially for those new to online learning. Toll-free 
telephone support should be available. Access to instructor support in regards to course 
content should also be available though another venue. 

Outcomes 

Recognition or credit for participating in modules and successfully completing them was 
important to learners. Developers should explore the issue of granting university credit 
or some other type of recognition. Evaluation of future courses offered by Health Canada 
is essential to ongoing success. Finally the recommendation was made that the report be 
made available to participants and course developers. 

CONCLUSION 

The evaluation focused on four criteria: design, content, process, and outcomes. Seven 
different sources were used to gather information for the evaluation. 

Forty-three public health professionals from across Canada actively participated in the 
pilot course entitled Introduction to Epidemiology. In general, the participants were 
satisfied with the experiences. However, there were frustrations with the technical 
aspects of the learning. There was question as to the relevance of the historical 
perspective in the unit, but the rest of the content was viewed as being of value. 
Learners also stated that they were able to transfer their learnings to real world 
applications. The original estimate of course duration was ten hours. Most people took 
three to four times longer than this, which caused stress for some. 

Participants were pleased with the immediate feedback received when doing quizzes and 
self-assessments. However, feedback from instructors took too long in many cases. This 
was at least partially due to the high student/teacher ratio and the fact the instructors 
were still devoting much time to developing course content. Pre- and post-tests caused 
anxiety to learners. In some cases they scored higher in pre-tests than post-tests. The 
discussion boards were not effectively used for student-to-student interaction as time 
was an issue; participants decided that if they were going to let something go, it turned 
out to be the discussions. 

Overall this pilot was seen as a success. The course needs to be refined, and 
recommendations can be used to improve future modules. Screen layout, navigation, 
organization, and presentation of content can be enhanced through the recommendations 
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given. Student orientation to online learning and system configuration would alleviate 
much frustration and time wasting efforts to figure things out. 

Time to complete the course should be 25 hours; therefore, the content must be reduced. 
The historical background might become optional and perhaps not be tested. Pre- and 
post-tests need to match content and learning objectives. 

Discussion groups need to be strengthened in order to bring about a feeling of a learning 
community. Class ratio for student/instructor should be kept to 1:20. Technical support 
should be addressed via a toll-free telephone and instructor support regarding content 
should be accessible through separate means. Students should know who to contact for 
technical advice and who to contact for content issues. 

Granting of credit for those who desire it should be explored. Participants should be able 
to access the Evaluation of the Skills Enhancement for Health Surveillance Pilot Online 
Module on Epidemiology Final Report. This strategy for providing professional 
development for public health employees is seen as a feasible and cost effective means to 
deliver quality training. Standards should be kept up through ongoing feedback and 
evaluation. 
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